Thanks for the @. I haven't reviewed the content of this PR closely so am limiting my comment only to the topic of the "Help Needed" part above.

I know nothing about quarto or how to integrate it with CRAN's vignette-building and vignette-checking. Maybe these examples of other packages using it successfully on CRAN could lead to some good reference implementations: https://github.com/search?q=org%3Acran%20path%3A*.qmd&type=code

I can at least tell you how we're addressing these things LightGBM, maybe it will be helpful.

In {lightgbm}, we have a lightweight system that has been working well for us (which you alluded to in #9906 (comment)). Here's how we do this there:

1. vignettes are written in .Rmd files with headmatter like the following, using {markdown} + {knitr}

output:
  markdown::html_format:
    options:
      toc: true
      number_sections: true
vignette: >
  %\VignetteIndexEntry{Basic Walkthrough}
  %\VignetteEngine{knitr::knitr}
  %\VignetteEncoding{UTF-8}

(code link)

2. The main docs site is built via Sphinx, and the Sphinx build builds a {pkgdown} site with the roxygen docs + vignettes + demos (code link)
3. those docs are integrated into the readthedocs site using an absolute hyperlink (code link)
4. these are linked from the "R API" nav item at https://lightgbm.readthedocs.io/en/latest/index.html, which directs to https://lightgbm.readthedocs.io/en/latest/R/reference/

It's a little awkward that the R docs are not technically part of the readthedocs site, but overall this has worked pretty well for us. Some notes:

• {markdown} + {knitr} is a lighter-weight combination than using {rmarkdown} or {quarto} (fewer required dependencies to install)
  • ref: [R-package] [ci] switch vignettes from 'rmarkdown' to 'markdown' microsoft/LightGBM#6258
• there are many examples of using .Rmd files + this combination on CRAN, so staying CRAN-compliant requires very little ongoing maintenance
• {pkgdown} is customizable enough to meet our needs, and it keeps up with changes to the RMarkdown format

Even if you don't pursue this specific mix, I do recommend not checking the .md version of the vignette into source control here. Instead, you could put any commands to generate it as part of the docs-site building into doc/conf.py or doc/Makefile. I shared a link above to how to run custom commands at build-time via conf.py. Here's an example of how to do it via the Makefile generate by sphinx: https://github.com/rapidsai/legate-boost/blob/ba062bf666845f82f5e8ab4690c97db4469ab42f/docs/Makefile#L17-L29

Thanks for the suggestions, but while LightGBM's solution works in the sense of providing rendered artifacts, I don't think it's an ideal solution here - the doc page URL differs from the Sphinx one and is not as easily indexable/searchable as an embedded .md or .ipynb file in the same sphinx page.

I've added a script and instructions to update the .md file here, although I'm not yet 100% sure that the .qmd vignette with the settings it has would render at CRAN without errors. Will hand over to @trivialfis from here. If this is still too complicated, maybe the logic could be switched to having the vignette as .Rmd and the script modifying the header and extension to .qmd to render the .md instead.

Still waiting from comments from @mayer79 if there are any.

Let me update the branch to prevent CI hang on mgpu tests. It's a AWS instance.

I don't think it's an ideal solution here - the doc page URL differs from the Sphinx one and is not as easily indexable/searchable as an embedded .md or .ipynb file in the same sphinx page.

Sure, those are good points. You have to balance whether those benefits you listed are worth the added build complexity, heavier set of dependencies, and increased risk of CRAN rejections. Not my call to make here in xgboost, and I don't have anything else to add.

I will take a look at the doc build tomorrow.

Hi, @david-cortes. May I ask what qdm provides that rmd doesn't in terms of generating md files?

For now, I think pkgdown with html output is the way to go. For a true integration with sphinx, we need a sphinx extension for the R domain, which is beyond the scope of XGBoost.

We can continue to use knitr for vignettes unless qmd provides extra capabilities that rmd doesn't have.

For information about sphinx domain, please see https://www.sphinx-doc.org/en/master/usage/domains/index.html .

Hi, @david-cortes. May I ask what qdm provides that rmd doesn't in terms of generating md files?

As far as I'm aware, .Rmd files are not compilable to .md and/or .ipynb which is what sphinx can support as embedded files to render, whereas .qmd can produce both formats.

Issue is: it appears to use jupyter to generate formats like .md and .ipynb - but I'm not familiar very familiar with either quarto or knitr so perhaps there might be other ways around it that I'm not aware of.

@trivialfis Example of how the result would look like:

Would be ideal to find a way to keep it looking like this.

Thank you for sharing!

We have a script to create markdown files, see changes in a #11166, specially around the makefile.

Although I'm not sure how knitr works.

I think the amount of time we need to spend to make a satisfactory result by various conversions will be about the same as writing a proper sphinx extension. As a result, I suggest we use pkgdown for now.

If you feel the sphinx integration is important for the R community, planning a future sphinx extension is better use of time than hacking various markdown flavors.

Please note that the knitr script doesn't create R api reference, while the pkgdown does.

Actually on a second look: turns out that knitr does support building to .md (but not to .ipynb unfortunately, so we couldn't add plots or similar), without IR. So no need to use .qmd.

Now the next question: could this be arranged in such a way that the .md file is built programmatically by the CI and commited to the repository?

If you believe the vegnittes should be rendered as sphinx doc, we can continue to use knitr and leave the API references to pkgdown

If you believe the vegnittes should be rendered as sphinx doc, we can continue to use knitr and leave the API references to pkgdown

Yes, that'd be ideal. But how would that be achieved?

Good question, I am sure I can enforce new commits to update the MD files, but I don't know if I can build the MD files automatically. In the referenced pr, the doc builder hangs.

Good question, I am sure I can enforce new commits to update the MD files, but I don't know if I can build the MD files automatically. In the referenced pr, the doc builder hangs.

That might not be a good idea - the .md would change for example whenever the random numbers produced differ, or when there are numerical differences in results between different CI machines, and so on.

Opened an issue for RTD readthedocs/readthedocs.org#11928

I will try to reproduce using their container image tomorrow.

@trivialfis In the CI, the Makefile under doc/R-package/ doesn't appear to be executed by the Makefile from doc.

I don't see any trace of it in the RTD log.

It's manual, I have never looked into it before. In my pr, I added some changes to the conf.py file to run it during Docs build.

Hopefully there's a better way to do it instead of burdening the RTD server

It's manual, I have never looked into it before. In my pr, I added some changes to the conf.py file to run it during Docs build.

Hopefully there's a better way to do it instead of burdening the RTD server

Ok, then let's leave the sphinx rendering for a different PR. I've removed all the references to .md files from this one, leaving only the part about adding a new vignette and examples.